Papers with autoregressive LLMs
PEFT-Bench: A Parameter-Efficient Fine-Tuning Methods Benchmark (2026.eacl-long)
Copied to clipboard
| Challenge: | Parameter-Efficient Fine-Tuning (PEFT) methods reduce the number of trainable parameters while maintaining strong downstream performance. |
| Approach: | They propose a unified benchmark for evaluating diverse PEFT methods on autoregressive LLMs. |
| Outcome: | The proposed methods reduce trainable parameters while maintaining strong downstream performance. |
Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on prompt engineering to extract sentence embeddings from large language models (LLMs) but these models are mostly decoder-only and the earlier tokens in the sentence cannot attend to the latter, resulting in biased encoding of sentence information and cascading effects on the final decoded token. |
| Approach: | They propose a plug-and-play and training-free technique that prepends each layer’s decoded sentence embedding to the beginning of the sentence in the next layer’ s input. |
| Outcome: | The proposed technique can significantly improve the performance of existing prompt-based sentence embedding methods across different LLMs while incurring negligible additional inference cost. |
MIDLM: Multi-Intent Detection with Bidirectional Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing models that use autoregressive architectures restrict the sharing of token information within a sentence. |
| Approach: | They propose a framework that integrates intent number detection and multi-intent selection to enable autoregressive LLMs to leverage bidirectional information awareness through post-training. |
| Outcome: | The proposed framework outperforms existing models and pretrained baselines in the multi-intent detection task. |
Okay, Let’s Do This! Modeling Event Coreference with Generated Rationales and Knowledge Distillation (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent work shows that generative large language models (LLMs) can be used to solve cross-document coreference problems. |
| Approach: | They propose rationale-oriented event clustering and knowledge distillation methods for event coreference scoring that leverage enriched information from the FTRs for improved CDCR. |
| Outcome: | The proposed model achieves SOTA B3 F1 on the ECB+ and GVC corpora without additional annotation or expensive document clustering. |
Improving the OOD Performance of Closed-Source LLMs on NLI Through Strategic Data Selection (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to improve robustness require changing the fine-tuning process or large-scale data augmentation, which are infeasible or cost prohibitive for closed-source models. |
| Approach: | They propose to prioritize more complex examples or replace existing training examples with LLM-generated data to improve performance on OOD NLI datasets. |
| Outcome: | The proposed methods improve performance on difficult OOD datasets while training with synthetic data leads to substantial improvements on easier OOD data. |
Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing autoregressive models generate tokens sequentially and are memory-bound, resulting in a memory-based inference stage that is memory-limited. |
| Approach: | They propose an approach to accelerate the inference speed of large language models with billions of parameters by integrating semi-autoregressive inference and speculative decoding capabilities. |
| Outcome: | The proposed approach has demonstrated inference speedups of 2.7x-4.0x on humanEval-X while maintaining output quality. |
TOP-Training: Target-Oriented Pretraining for Medical Extractive Question Answering (2025.coling-main)
Copied to clipboard
| Challenge: | e-health records underscore the growing significance of information extraction (IE) from these datasets. |
| Approach: | They propose a target-oriented pre-training paradigm for extractive question-answering in the medical domain . TOP-Training moves one step further than popular domain-oriented fine-tuning . |
| Outcome: | The proposed method improves on the Medical-EQA benchmarks. |
Differentially Private Knowledge Distillation via Synthetic Text Generation (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language models (LLMs) are achieving state-of-the-art performance in many downstream tasks, but data privacy is a major challenge for practitioners. |
| Approach: | They propose a differentially private knowledge distillation algorithm that exploits the knowledge of a teacher LLM and a student's output distribution. |
| Outcome: | The proposed algorithm significantly improves the utility over baselines on the Big Patent dataset, with strong privacy parameters, =2. |
FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping (2024.emnlp-main)
Copied to clipboard
| Challenge: | Autoregressive Large Language Models (LLMs) are omnipresent but typically come with a substantial model size. |
| Approach: | They propose a novel fine-grained skip strategy for autoregressive large language models . they observe the saturation of computationally expensive feed-forward blocks of LLMs . |
| Outcome: | The proposed method can skip 25-30% of FFN blocks with marginal change in performance on knowledge-intensive generation tasks. |
DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreak (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are susceptible to generating harmful content when prompted with carefully crafted inputs, a vulnerability known as LLM jailbreaking. |
| Approach: | They propose an end-to-end generative approach for jailbreak rewriting inspired by diffusion models that uses a sequence-tosequence (seq2sequ) diffusion model as a generator, conditioning on the original prompt and guiding the denoising process with a novel attack loss. |
| Outcome: | Experiments on Advbench and Harmbench show that the proposed method outperforms autoregressive jailbreak models across evaluation metrics including ASR, fluency, diversity and diversity. |
EvoMD-LLM: Learning the Language of Species Evolution in Reactive Molecular Dynamics (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing models operate on static molecular representations or rely on external tools for reasoning. |
| Approach: | They propose a framework that reformulates species-level molecular dynamics as a symbolic temporal language modeling problem. |
| Outcome: | The proposed model outperforms neural networks and language-based baselines on multiple temporal prediction tasks and generates plausible interpretations of reaction dynamics. |